On Weak Base Hypotheses and Their Implications

1 2 When studying the training error and the prediction error for boosting, it is often assumed that the hypotheses returned by the base learner are weakly accurate, or are able to beat a random guesser by a certain amount of diierence. It is has been an open question how much this diierence can be, whether it will eventually disappear in the boosting process or be bounded by a nite amount see, e.g., Schapire et al. (1998), Section 3]. This question is crucial for both the behavior of the training error and prediction error. In this paper we solve the problem in the aarmative, i.e., the amount of improvement over the random guesser will be at least a nite amount for all possible sample realizations and for most of the commonly used base hypotheses. This has a number of implications on the prediction error, e.g., that boosting forever may not be good and regularization may not be unnecessary. The problem is solved by rst considering an analog of AdaBoost in regression, where we study similar properties and nd that for a good performance, one cannot hope to avoid regularization by just adopting the boosting device to regression. Examples are given where at least some regularized versions of the regression and classiication boosting algorithms can be proved to perform better asymptotically than running the unmodiied algorithms forever.